Skip to content

feat(prose-tests): the framework — cases, worlds, runner, skill#545

Open
leeovery wants to merge 2 commits into
design/prose-testsfrom
prose-tests/framework
Open

feat(prose-tests): the framework — cases, worlds, runner, skill#545
leeovery wants to merge 2 commits into
design/prose-testsfrom
prose-tests/framework

Conversation

@leeovery

@leeovery leeovery commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Summary

  • The deterministic layer of the prose-test framework (stage 2 of design/prose-tests.md): case parser + corpus validation (tests/prose/lib/cases.cjs), fixture recipes run under a frozen-clock preload with committed golden snapshots (lib/fixtures.cjs, lib/fake-clock.cjs), the world builder (lib/world.cjs — installed-layout skills, hermetic git, keyword-only knowledge store re-derived at materialise), and the runner CLI (run.cjs: list · select · world · prompt · grade · snap · verify · destroy).
  • P4 enforced structurally: walker prompts are machine-assembled by run.cjs prompt and never contain expect: claims; grade runs state assertions deterministically via the world's own engine and emits routing claims for the grader agent only.
  • Two token-free CI suites join npm test: corpus validation (parse, scoped paths, anchors, assertion grammar) and fixture golden checks (every snapshot rebuilds byte-identical from its recipe — drift lands as a reviewable snapshot diff, never silently). Plus the /prose-test dev skill (Sonnet walks, Opus confirms failures, quoted evidence required) and the base fixture with two smoke cases.

Test plan

  • npm test green (1683 tests, includes both new suites; base fixture rebuild ~2.9s).
  • Determinism proven: node tests/prose/run.cjs verify run twice, byte-identical both times.
  • Full pipe executed live: world built, engine boot run in-world (knowledge ready, zero migrations), deterministic grade passed, walker prompt verified expect-free, world destroyed.

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545 👈 current
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571

leeovery and others added 2 commits July 24, 2026 16:52
Stage 2 of design/prose-tests.md. The deterministic layer in code:
case parser + corpus validation, fixture recipes under a frozen clock
with golden snapshots (rebuild-compare in npm test), the world builder
(installed-layout skills, hermetic git, keyword-only knowledge store),
and the runner whose `prompt` command is the P4 boundary — walker
prompts never carry expects. The /prose-test dev skill owns the model
layer: Sonnet walks, Opus confirms failures, quoted evidence required.
First fixture (base) proven byte-deterministic; two smoke cases
exercised the full pipe end-to-end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review finding (Lee): coordinate-phrased cases (step numbers, numbered
heading anchors) break on cosmetic renumbering — failure for the wrong
reason. Anchors become substring fragments (includes-matched, still red
token-free when the heading vanishes), the smoke cases are rewritten
purely behaviourally, and the authoring rule lands in the README.
Prose test hooks considered and parked in the design log: the walker
binds semantically, so shipped-prose plumbing isn't earning its keep —
revisit on evidence at the failure harvest.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant